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Objectives 

Establish  a  view  of  the  acquirer  and  supplier/contractor 
roles  and  responsibilities. 

Show  how  measurement  and  analysis  skills  for  internal 
development  can  be  recast  for  acquisition  and  contracting 
environments. 

Address  two  prevalent  questions  in  the  acquisition 
community: 

•  How  can  measurement  be  used  to  improve 
requirements-related  processes? 

•  How  can  we  conduct  causal  analysis  when  we  no 
longer  control  the  collection  processes  and/or  data? 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Terms  and  Usage 

We  use  the  terms  “acquisition”  and  “contracting” 
interchangeably  throughout  this  tutorial. 


In  addition,  the  terms  “contractor”  and  “supplier”  are  used 
interchangeably.  The  term  “developer,”  in  the  context  of 
this  tutorial,  is  used  to  describe  a  contractor. 
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Trends  in  Outsourcing  1 

From  Gartner  Group  (2002) 

•  one  out  of  every  10  jobs  with  U.S. -based  information 
technology  vendors  and  service  providers  will  be  exported 

•  more  than  80  percent  of  corporate  boards  of  directors  will 
have  considered  offshore  outsourcing 

•  40  percent  of  corporations  will  have  finished  an  outsourcing 
pilot  program  or  be  actively  involved  in  outsourcing 
technology  services 

From  Forrester  Research 

•  offshore  outsourcing  will  account  for  28%  of  IT  budgets  in 
Europe  and  the  U.S.  by  2004 

•  offshore  IT  workers  will  go  from  360,000  (in  2002)  to  more 
than  1  million  in  2005 


[www.rosourcing.com],  [robb  02],  [diana  03] 

©2003  by  Carnegie  Mellon  University  Version  1.0 


page  6 


Carnegie  Mellon 

Software  Engineering  Institute 


Trends  in  Outsourcing  2 

From  Michael  F.  Corbett  &  Associates: 

•  Offshore  outsourcing  is  just  one  small  part  of  a  (US)$5  trillion 
global  outsourcing  market. 

•  This  market  is  growing  by  more  than  15  percent  per  year,  and 
the  offshore  component  is  certainly  among  the  fastest  growing 

•  For  U.S.  IT  professionals,  this  probably  means  that  their 
future  success  will  come  from  moving  up  the  IT  value  chain 

From  Ovum  research 

•  The  outlook  for  the  future  is  more  offshore  outsourcing,  but 
not  at  the  levels  predicted  by  other  analysts  in  this  area 


[www.rosourcing.com],  [robb  02],  [diana  03] 
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Why  Do  Organizations  Outsource? 

Top  10  Reasons  from  The  Outsourcing  Institute: 

•  Reduce  and  control  operating  costs 

•  Improve  company  focus 

•  Gain  access  to  world-class  capabilities 

•  Free  internal  resources  for  other  purposes 

•  Resources  are  not  available  internally 

•  Accelerate  reengineering  benefits 

•  Function  difficult  to  manage/out  of  control 

•  Make  capital  funds  available 

•  Share  risks 

•  Cash  infusion 


[www.rosourcing.com] 
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The  Supplier  Landscape  1 

Contractor  dimensions: 

•  geography 

•  style 

•  maturity 

•  processes 

Examples  include  the  following: 

•  domestic  development  groups 

•  offshore  development  groups 

•  dedicated  offshore  development  centers 

•  off  the  shelf,  COTS  products 

•  systems  integrators 

•  open  source 

•  rational 

•  PSP/TSP 
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The  Supplier  Landscape  2 

From  Forester  Research 

•  88%  of  the  firms  looking  overseas  for  services  claim  to 
get  better  value  for  their  money  off  shore. 

•  71%  said  offshore  workers  did  better  quality  work. 


[www.rosourcing.com] 
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Contracting  Challenges  1 

From  Software  Magazine  in  2001 : 

•  23%  of  software  projects  are  cancelled 

•  Cost  growth  averages  45% 

•  Schedule  growth  averages  67% 

•  Average  final  product  will  include  only  67%  of  its  requirements 

•  Only  28%  of  projects  finish  on  schedule  and  within  budget 

Cited  by  a  sampling  of  Army  Acquisition  Managers 

•  The  majority  of  problems  and  risks  affecting  acquisition 
problems  resides  “somewhat”  with  the  following: 

-  factors  outside  the  control  of  acquirers  and  developers 

-  acquisition  program  policies  and  processes 

-  contracting  processes 

-  the  contractor’s  development  process 


[ASSIP  03],  [SWM  01] 
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Contracting  Challenges  2 

Cited  by  a  sampling  of  Army  Acquisition  Managers 

•  The  top  problem  areas  include 

-  requirements  management  (selected  by  63%) 

-  project  management  (22%) 

-  contractor  processes  (22%) 

-  unstable  funding  (21%) 

From  a  recent  presentation  on  component  technology 

•  contractor  qualifications  (Mitigation:  CMMI) 

•  requirements  definition  (Mitigation:  dose  partnerships) 

•  engineering  acceptance  (Mitigation:  process  analysis) 


[ASS IP  03],  [Scherlis  03] 
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Measurement  Challenges 

From  interviews  of  several  acquisition  management 
personnel: 

•  “Measurement”  is  not  a  troublesome  issue  in  itself; 
however,  getting  consistent,  meaningful  data  and 
understanding  how  to  use  data  is  a  high  priority  and 
concern. 

•  There  is  a  tremendous  need  for  progress  measures 
that  can  be  used  for  timely  warning  of  major  program 
disasters. 


[C-M-H  03] 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  13 


Carnegie  Mellon 

Software  Engineering  Institute 


Measures  in  Practice  1 

In  a  recent  survey,  a  sampling  of  Army  Acquisition  Managers 
affirmed  the  following 

•  83%  based  planning  estimates  on  historical  data 

•  79%  defined  quantitative  objectives  for  acquired  products  and 
services 

•  81%  used  metrics  as  an  input  to  decision  making 

•  75%  measured  and  controlled  project  cost  and  schedule 

•  50%  recorded  data  in  organizational  measurement  repository 

•  78%  had  sufficient  insight  into  the  contractor’s  software 
engineering  effort  to  ensure  project  is  managed  and  controlled 
and  complies  with  contract  requirements 

•  78%  appraised  the  quality  of  the  contractor’s  process, 
performance,  products,  and  services  throughout  the  contract 
to  identify  risks  and  take  appropriate  action 


[ASS IP  03] 
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Measures  in  Practice  2 

The  surveyed  Army  Acquisition  Managers  use  these 
measures  to  track  project  status: 


Schedule  Cost  Development  Manpower  Requirements 

Progress  Stability 


[ASS IP  03] 
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What  Does  This  Mean? 

Issues  in  contracting  are  complex  and  multidimensional. 

•  Requirements  management  is  a  problem  area  that 
frequently  is  not  well  measured. 

•  Project  monitoring  and  oversight  is  fairly  well  measured, 
but  the  related  analysis  may  not  be  mastered. 

•  Organizations  may  often  measure  what  they  know  how 
to  measure,  but  not  necessarily  measure  all  that  is 
needed  to  be  successful. 

How  does  this  compare  to  your  experience? 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Responsibility  and  Authority 

Measuring  project  and  product  success  is  the  same  whether  the 
project  is  internal  or  contracted: 

•  on  schedule 

•  at  cost 

•  with  required  functionality 

•  without  defects 

The  acquiring  program  manager’s  “circle  of  influence”  and  “circle  of 
control”  is  different  than  the  development  project  manager’s. 

•  development  project  manager  addresses  the  daily  details  of 
project  execution 

•  acquisition  program  manager  defines  and  executes  a  new  set  of 
processes 

•  acquisition  program  manager  should  leverage  development 
knowledge  to  manage  the  contract  methodically,  rationally,  and 
knowledgeably 
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Roles  and  Information  Exchange 


Contractual  Handshake 
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Acquisition  Measurement  Themes 

Project  Management 

•  project  execution 

•  contract  relationship 

Product  Life  Cycle  &  Performance 

•  product  planning 

•  product  development 

•  deployment 

•  maintenance 

Process  &  Organizational  Infrastructure 

•  process  definition  and  execution 

•  relationship  management 
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Measuring  Project,  Product,  Process 


Processes  -s 


Acquirer 


Pre-award 

activities 

Post-award 

activities 


Project 

•Schedule 

(status,  projection,  trend) 
•Cost 

(status,  projection,  trend) 
•Requirements  satisfaction 


Contractual 

Handshake 


Exchange  of 
indicators  / 
information 
for  tracking, 
monitoring, 
direction,  etc, 


Supplier 


Develop, 

Customize, 

Integrate 

•  systems 

•  software 

•  COTS 


Products 


Relationship 

•Roles  (changes) 

•Invoicing  (payment) 


•Supplier  Produced 
•Quality  (amount  of  rework) 

•Acquisition  Organization  Produced 
•Quality  (amount  of  rework) 
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Responsibilities  Prior  to  Contract 
Award 

Scope  definition 

Vendor  selection 

•  technical  capabilities 

-  proposed  scope 

•  process  capabilities 

-  predictable,  productive  performance 

-  ability  to  deal  with  change 

•  financial  capabilities 

Contract  negotiation 

•  quality  management  metrics 

•  change  management 

•  managing  &  monitoring  the  relationship 
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Responsibilities  After  Contract 
Award 


Contractor 


Deliverables 


ACQUIRER 


Documents 

-  SRD 

-  SDP 

-  Measurement 
Plan 

-  SDD 

Status  Reports 

-  Schedule 

-  Cost 

-  Testing 

Final  Product 


Acquirer  Responsibilities 
(Post-Contract  Award) 


Evaluate  Quality  of 
Deliverables 

Monitor  and  Oversight 

-  Schedule  &  Progress 

-  Resources  &  Costs 

-  Developer’s  Processes 


£. 
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Monitor  &  Oversight 


Measurable  Results  (Examples) 

•  contractor  effort  actual  vs.  plan 

•  contractor  schedule  actual  vs.  plan 

•  defects  reported 

•  description,  severity,  class,  type 

•  size,  complexity  of  the  work  product 


Indicators 
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Evaluate  Quality  of  Deliverables 
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Success  Factors 

To  make  this  work  you  need: 

•  technical  capabilities 

-  integration,  validation,  deployment 

•  process  capabilities 

-  project  management,  QA,  change  control 

•  domain  knowledge 

-  product  uses,  stakeholders,  quality  goals 

•  relationship  management 

-  contracting,  change  management,  roles,  payment, 
relationship  reviews.... 

And  measurement  to  see  that  these  things  are  working 
well. 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Adapting  CMMI  for  Acquisition 

In  addition  to  establishing  these  Process  Areas  (PAs) 

•  Supplier  Agreement  Management 

•  Integrated  Supplier  Management 

You  may  also  need  to  use  these  PAs  for  your  acquisition 
processes  and  extend  them  to  include  your  supplier: 

•  Requirements  Management,  Development 

•  Integrated  Teaming 

•  Decision  Analysis  and  Resolution 

•  Organizational  Environment  for  Integration 

•  Organizational  Process  Performance 

•  Quantitative  Project  Management 

•  Causal  Analysis  and  Resolution 

•  Risk  Management 

•  Project  Monitoring  and  Control 

•  Verification  &  Validation 

•  Configuration  Management 

•  Measurement  and  Analysis 
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SA-CMM  Key  Process  Areas 


Level 

Focus 

Key  Process  Areas 

5 

Optimizing 

Continuous 

process 

improvement 

Acquisition  Innovation  Management 
Continuous  Process  Improvement 

Higher 

Quality 

4 

Quantitative 

Quantitative 

management 

Quantitative  Acquisition  Management 
Quantitative  Process  Management 

Productivity 

Lower  Risk 

3 

Defined 

Process 

standardization 

Training  Program  Management 
Acquisition  Risk  Management 

Contract  Performance  Management 
Project  Performance  Management 

User  requirements 

Process  Definition  and  Maintenance 

2 

Repeatable 

Basic 

project 

management 

Transition  to  Support 

Evaluation 

Contract  Tracking  and  Oversight 
Project  Management 

Requirements  Development  and  Mgt. 
Solicitation 

Software  Acquisition  Planning 

Higher  Risk 

Rework 

1 

Initial 

Competent  people  and  heroics 
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Relation  to  CMMI  PAs 


CMMI  Process  Area  SA-CMM  KPA 


Software  Acquisition  Planning 
Project  Management 
Solicitation 

Contract  Tracking  and  Oversight 
Requirements  Development  and 
Management 

Validation 

Configuration  Management 
Decision  Analysis  and  Resolution 
Organizational  Training 


Project  Planning  ◄ - - - 

Project  Monitoring  And  Control  +- 
Integrated  Supplier  Management^ 
Risk  Management 
Requirements  Development 
Requirements  Management 
Verification 


[Ferguson  03] 
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Maturity  Matching  Considerations 


A 


■H 

'C 

3 
+-» 
ro 

E 

!5 
(0 
Q. 

(0 
O 

I 

Management 
Capability 
Level 

[Barbour  0 
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Mismatch 

•  mature  buyer 
must  mentor  low 
maturity 
developer 

•  outcome  not 
predictable 


Disaster 

•  constant  crises 

•  no  req’s  mgt. 

•  no  risk  mgt. 

•  no  discipline 

•  no  process.  . . 

•  no  product 


Matched  Team 

•match  of  skills,  maturity 
•team  risk  approach 
•execution  to  plan 
•measurable  performance 
•quantitative  management 

hiahest  probability  of 


success 


Mismatch 

•  “Customer  is 
always  right” 
hurts. 

•  Customer 
encourages 
“short  cuts.” 


capability/maturity 


Supplier  (developer) 
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Focusing  In 

Key  points: 

•  trends  in  contracting 

•  common  problems  and  issues 
faced  when  contracting 

•  common  view  of  the  roles  and 
responsibilities  of  an  acquirer 

•  role  of  reference  models 

What’s  in  sight: 

•  measurement  and  analysis 
techniques 

In  the  distance: 

•  an  illustration  of  these  techniques 
at  work 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Benefits  of  Using  Measures 

Measurement  by  itself  does  not  control  or  improve;  it 
gives  insight  for  objectively  planning,  managing,  and 
communicating. 

•  historical  data  help  us  predict  and  plan 

•  actual  versus  plan  data  help  us  determine  progress 
and  support  decision  making 

•  analyzing  trends  helps  us  identify  and  focus  on 
problem  areas 

•  project  data  provide  a  basis  for  objective 
communication 
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Measurement  in  CMMI  Process  Areas 

Project  Management 

•Project  Planning,  Project  Monitoring  and  Control,  Software 
Acquisition  Management 

•Integrated  Project  Management,  Risk  Management,  Integrated 
Supplier  Management 

•  Quantitative  Project  Management 

Process  Management 

•Organization  Process  Focus,  Organization  Process  Definition 
•Organization  Process  Performance 
•Organization  Innovation  and  Deployment 

Engineering  --  All 

Support 

•  Measurement  and  Analysis,  Process  and  Product  Quality 
Assurance 

•Decision  Analysis  and  Resolution 
•Causal  Analysis  and  Resolution 
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Measurement  in  CMMI  Generic  Practices 

“Monitor  and  control  the  process  against  the  plan  and  take 
appropriate  corrective  action.”  (GP2.8) 

“Collect  work  products,  measures,  measurement  results, 
and  improvement  information  derived  from  planning  and 
performing  the  process  to  support  the  future  use  and 
improvement  of  the  organization’s  processes  and  process 
assets.”  (GP3.2) 

Two  uses  of  measurement: 

•  project  management 

•  process  improvement 

As  the  organization  matures,  the  sophistication  and  uses  of 
measurement  increase. 
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Measurement  in  SA-CMM 

Maturity  Levels  2-5 

•  status  of 

-  processes 

-  products 

Maturity  Levels  4-5 

•  effectiveness  of 

-  processes 

-  products 
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Acquisition  Enterprise  Measurement 

Execution  of  a  contracted  project  also  involves 

•  legal  processes 

•  financial  processes 


While  this  tutorial  does  not  explore  these  aspects  of 
contracting,  each  aspect  is  measurable  and  can  be 
quantitatively  managed. 
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Sources  for  Measures 

Goal-Driven  (Software)  Measurement  (GDM) 


Goals— ►Questions  —►Indicators  -►Measures  (GQIM) 
USER  DEFINES  INDICATORS  &  MEASURES 

Based  On: 

•  what’s  needed  to  manage  the  User’s  goals 

•  decisions  and  decision  criteria  related  to  managing 
the  user’s  goals 


Practical  Software  &  Systems  Measurement 


Common  — 
Issue 

Area 

-►  Measurement 
Category 

— ►  Measures 

PREDEFINED 

PREDEFINED 

PREDEFINED 
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Goal-Driven  Measurement  (GDM) 

When  using  goal-driven  measurement,  the  primary 
question  is  NOT: 

“What  metrics  should  I  use?” 

rather,  it  is: 

“What  do  I  want  to  know  or  learn?” 

“What  decision  do  I  want  to  make?” 


Goal-driven  measurement  is  NOT  based  on  a 
predefined  set  of  metrics. 


[GQIM  96] 
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Goal-Driven 

Software 

Measurement 

(GDM) 

[GQIM  96] 


SLOC  Staff-hours  Trouble  Reports  Milestone  dates 


definition 

checklist 

_ Ef 

_ Ef 

_ Ef 

_ Ef 

_ Ef 


Indicator  Template 

Objective  - 

Question  - 


Inputs  — 
Algorithm  — 
Assumptions 


Infrastructure 

Assessment 


>=> 
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Practical  Software  &  Systems 
Measurement  (PSM) 

This  measurement  process  is  funded  by  the  DoD  and  is 
freely  available  at  http://www.psmsc.com. 

PSM  process  identifies  project-specific  issues: 

•  issues  grouped  into  common  software  issue  areas 

•  measurement  categories  correspond  to  issue  areas 

•  each  measurement  category  has  a  candidate  set  of 
proven  measures 

Measures  are  selected  based  on  availability, 
environment,  and  other  factors. 
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PSM  Common  Software  Issues  - 
Measurement  Categories 


Schedule  and  Progress 

-  Milestones  Performance 

-  Work  Unit  Progress 

-  Incremental  Capability 

Product  Size  and  Stability 

-  Product  Size  and  Stability 

-  Functional  Size  and  Stability 

Process  Performance 

-  Process  Compliance 

-  Process  Efficiency 

-  Process  Effectiveness 

Customer  Satisfaction 

-  Customer  Feedback 

-  Customer  Support 


Resources  and  Cost 

-  Personnel 

-  Financial  Performance 

-  Environment  Availability 

Product  Quality 

-  Functional  Correctness 

-  Supportability  -  Maintainability 

-  Efficiency 

-  Portability 

-  Usability 

-  Dependability  -  Reliability 

Technical  Effectiveness 

-  Technology  Suitability 

-  Impact 

-  Technology  Volatility  [psM  00] 
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Modified  Indicator  Template 


Indicator  Name/Title 

Objective  - 

Questions  - 

Visua 

A 


Date 


Establish 
Measurement 
Objectives 


Communicate 
Results 


^00 

so 

50 

f40 

20 


flfl 


Perspective 

Input(s) 

Data  Elements 
Definitions 

Data  Collection 
How 

When/How  Ofte 
By  Whom 
Form(s) 

Data  Reporting 

Responsibility 
for  Reporting 

By/To  Whom 

How  Often 


r 

Specify 

/leasure 

1 

L 

J 

Assumptions  - 

Algorithm 
Interpretation  — 

Probing  Questions 
Analysis  — 

Evolution 

Feedback  Guidelines 
X-reference  - 


A 

^ — 

i. 

f  i 

L  Pr< 

Specify  ‘ 

analysis 

ocedures 

Analy 

Dat 

rze  ] 

i 

r 

Additional  Modifications  by  clients 

•  streamlined  data  collection  & 
reporting  sections  using 
“swimlane”  diagrams 

•  Addition  of  “corrective  action 
guidelines” 

•  Subprocess  selection  (for 
CMMI) 

[GQIM],  [DZ  02] 
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Indicator  Classifications  gq(i)m 

GQ(I)M 


Roll-up  For 
Higher  Management 


Goal 


hi 


Reporting  Periods 


r  ^ 

Analysis  Indicators 

What  are  results  of 
specific  tasks? 


i 


Strategy  to 
accomplish 
the  goal 


_ 


I 


Me 


PSM,  GQ(I)M 


Tasks  to  Accomplish 
goal 


Task  1 
Task  2 
Task  3 

Task  n 
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Success 

Criteria 


PSM 


Success  Indicators 

Have  the  goals  been 
achieved?  What  is  the 
impact  of  the  tactics? 


GQ(I)M 


For 

Project  Manager 


Reporting  Periods 


Roll-up  For 
Higher  Management 


hi 


Actual 

V 

Planned 

&L ■  ■  ■  ■ 


Reporting  Periods 


Progress  Indicators 

How  well  are  plans  proceeding? 
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Data  Analysis  Dynamics 


Getting  Started 

•  Identify  the  goals 

•  Black  box  process  view 

•  Is  the  data  right? 

•  Do  I  have  the  right  data? 

Decision  point: 

•  If  the  data  is  not  perfect,  do  I 
move  forward  or  obtain  better 
data? 

Initial  Evaluation 

•  What  should  the  data  look  like? 

•  What  does  the  data  look  like? 

•  Can  I  characterize  the  process, 
product,  problem? 


Decision  point: 

•  Can  I  address  my  goals 
right  now? 

•  Or  is  additional  analysis 
necessary?  at  the  same  or 
deeper  level  of  detail? 

•  Can  I  move  forward? 

Moving  Forward 

•  Further  evaluation 

•  Decompose  data,  process 

Decision  point: 

•  Do  I  take  action? 

•  What  action  do  I  take? 

Repeat  until  root  cause  found,  at 
target  with  desired  variation 


[DAD  03] 
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Performance  Analysis  Model 
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Performance  Analysis  Checklist  1 

Single  indicator  issues: 

•  Do  actual  trends  correspond  to  planned  trends,  such  as 
progress,  growth,  and  expenditures?  How  big  is  the 
variance? 

•  Does  the  variance  appear  to  be  gradually  growing  each 
month? 

•  Are  actual  values  exceeding  planned  limits,  such  as 
open  defects,  changes,  and  resource  utilization? 

•  Are  outliers  or  other  anomalies  affecting  the  results? 


[PSM  00] 
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Performance  Analysis  Checklist  2 

Integrated  indicator  issues: 

•  Is  the  source  of  the  problem  evident? 

-  Change  in  functionality,  unplanned  rework,  etc. 

•  Are  growing  problems  in  one  area  a  leading  indicator  of 
other  problems  later  in  the  project? 

-  Requirements  creep  impact  on  schedule 

•  Do  multiple  indicators  lead  to  similar  conclusions? 

-  Lack  of  progress  correlates  with  low  staffing 

•  Does  other  project  information  contradict  performance 
results? 

-  Milestones  being  met  but  open  defect  counts  are 
increasing 


[PSM  00] 
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Focusing  In 

Earlier: 

•  trends,  roles,  models 

Key  Points: 

•  measurement  in  maturity 
models 

•  three  indicator  types:  success, 
progress,  analysis 

•  comparing  PSM  and  GQIM 

•  Performance  Analysis  Model 

What’s  in  sight: 

•  an  illustration  of  these 
methods  at  work 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Illustration 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Composite  Illustration* 

This  illustration  is  based  on  an  organization  that  is 

•  maintaining  an  existing  product,  a  blend  of  COTS,  and 
internally  developed  code 

•  pursuing  the  acquisition  of  a  replacement  product 

Their  acquisition  includes  two  contracts: 

•  requirements  development 

•  product  design,  code,  and  test 

This  illustration  will  focus  on 

•  evaluating  requirements  document  quality  (contract  1) 

•  analyzing  project  execution  data  (contract  2) 

It  will  briefly  highlight  other  aspects  of  acquisition  measurement. 

*This  illustration  is  a  composite  of  two  projects.  Aspects  from  other  projects  have  been  interwoven 
for  demonstration  purposes. 
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Roles  and  Information  Exchange 


Contractual  Handshake 
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Sr.  Mgmt  dashboard 

•  quality  trends 

•  selected  project  EV  data 


Middle  Mgmt  dashboard 

•  system  documentation  and 
testing 


t 


Analysis  Indicators 

Reqts  completeness  - 
original,  at  inspection, 
approved  (for  contract  1 ) 


Goal: 

Establish  Acquisition 
Processes 

l 


Sr.  Mgmt  scorecard  ; 
Middle  Mgmt  dashboard 


Strategy  to  accomplish  goal 

•  Reference  models:  CMMI,  SA  CMM, 

ieee/iso  12207 

•  Leverage  CMMI  capabilities  built  in 
engineering:  MA,  REQM,  RD,  CAR 

•  Aim  for  CMMI  capability  in  selected  PAs: 
SAM,  DAR,  RSK,  PP/PMC,  CM,  PPQA 

•  Reference  all  SA-CMM  Level  2  kPAs, 
noting  overlaps  with  CMMI 


! _ 

Tasks  to  Accomplish  goal 

•  Implement  requirements  management 
process 

•  Tailor  existing  project  monitoring  processes 
for  acquisition  managers 


Success  Indicators 

process  owners,  training, 

CM,  and  documentation 
(future:  procedural  adherence) 


Middle  Mgmt  Dashboard 

•  selected  SPI  plan  EV  data 


Progress  Indicators 

start,  finish  dates 
with  progress  noted 
(move  toward  EV) 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Evaluate  Quality  of  Deliverables 


Carnegie  Mellon 

Software  Engineering  Institute 

Requirements  Development  & 
Management  (SA-CMM  RDM) 

Purpose: 

To  establish  a  common  understanding  of  the  software 
requirements  by  the  acquisition  project  team,  the  end  user, 
and  the  contractor. 

•  includes  both  technical  and  non-technical  requirements 

•  involves  development  of  the  requirements  and 
management  of  any  changes 

•  starts  with  description  of  an  operational  need  and  ends 
with  transfer  of  responsibility  to  the  maintainer 
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RDM  -  Measurement  Opportunities 


Process  Measures 
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Illustration:  Reqts  Process  Flow 


End  User  Acquirer  Contractor  1  Contractor  2 
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Requirements  Process  Measures 

Process  Measures 

•  effort  expended 

•  funds  expended 

•  progress  toward  completion 

•  completion  of  milestones 

•  number  of  change  requests  processed  (post¬ 
development) 

For  the  contractor,  these  are  measures  of  development 
process. 

For  the  acquirer,  these  are  measures  of  the  inspection 
process. 
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Requirements  Document 
Measures  and  Evaluation  Criteria 

“Inch”  or  “thickness”  Criterion 

•  Document  is  at  least  three  inches  thick 

“Drop  it”  or  “Thud”  Criterion 

•  Related  to  inch  criterion 

•  Specific  level  of  sound  before 

Format 

•  Pretty  pictures 

•  In  color 
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Requirements  Document 
Effective  Evaluation  Criteria 

Examples  of  measurements  for  evaluation  criteria 

•  completeness: 

-  “TBD”  requirements; 

-  product  performance  measures  included 

•  consistency: 

-  no  conflicts  across  document  sections 

•  clarity: 

-  growth  in  issues, 

-  presence  of  ambiguous  language  or  words  with  many 
meanings. 

•  conformity: 

-  meets  stated  criteria,  constraints 

•  correctness: 

-  all  data  fields  in  valid  ranges 
Contract  should  contain  evaluation  criteria. 
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Illustration:  Requirements  Indicator 


Requirements  Completeness 

by  function,  as  process  proceeds 


□  Complete  H  To  be  determined 
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Practical  Issues 

The  organization  or  program/project  office  may  have 
several  barriers  to  effective  document  inspection,  such  as 

•  insufficient  quantity/availability  of  personnel 

•  insufficient  technical  or  domain  knowledge 

•  schedule  constraints 

Example: 

•  If  you  have  a  300  page  requirements  document  and 
typically  inspect  at  a  rate  of  2  hrs/page,  are  there 
resources  available  to  invest  600  hours  to  inspect  that 
document? 
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Advancing  the  State  of 
Requirements  Product  Measures 


Manual  Automated 


Lengthy,  labor  intensive  process  Reduce  cycle  time  and  effort 

while  producing  better  results 

than  possible  with  tedious 
manual  review 


Examples  of  Tools: 

•  Quality  Analyzer  for  Requirements  Specifications  (QuARS) 

-  Lexical,  syntactic  and  semantic  analyses  of  requirements 
•Automated  Requirements  Measurement  (ARM) 
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Quality  Analyzer  for 
Requirements  Specification 

How  does  it  work? 

•  natural  language  analysis  of  requirements  text 

•  lexical:  vague,  weak,  optional,  subjective,  other  terms 

•  syntactic:  multiple,  implicit,  under  specified  statements 

•  semantic: 

-  allows  screening  for  consistency,  completeness,  etc. 

-  arbitrary  combinations  of  domains,  components, 
functionality,  product  quality  attributes,  and  so  on 
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Automated  Requirements 
Measurement  (ARM) 

Checks  for  desirable  requirements  characteristics  such  as: 

•  complete:  precisely  define  all  real  world  situations 

•  consistent:  no  conflict  between  individual  requirements 

•  correct 

•  modifiable 

•  ranked 

•  traceable 

•  unambiguous:  can  only  be  interpreted  one  way 

•  understandable:  meaning  of  each  of  its  statements  is 
easily  grasped  by  all  of  its  readers 

•  verifiable 

•  validatable:  by  individuals  and  organizations  having 
vested  interest 

•  testable 
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Focusing  In 

Earlier: 

•  trends,  roles,  models 

•  measurement  methods 

Key  Points: 

•  quality  of  deliverables 

•  effective  evaluation  criteria 

•  measuring  requirements 
development  (contract  1) 

•  tools  for  analyzing  requirements 


What’s  in  sight: 

•monitoring  and  oversight:  evaluating  a  schedule  slip 
(contract  2) 

•What  would  YOU  include  in  the  contract? 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  70 


Carnegie  Mellon 

Software  Engineering  Institute 


Monitoring  &  Oversight 

Contract  #2  has  been  awarded. 

•  supplier  is  developing  the  product  in  two  builds 

The  contractor  has  just  notified  you  that  the  project  has 
both  cost  and  schedule  slippage. 

What  do  you  do? 
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Performance  Analysis  Model 

Use  model  to  guide  analysis. 

•  Step  1:  Confirm  Problem  (Cost  &  Schedule  Slippage) 
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Schedule  &  Progress  Indicators 


JFMAMJ  JASONDJFMAMJ  JASON  D 


2002  2003 


Life  cycle  Activities 

Est.  Requirements 
Architecture 
Build  1 

Design 

Assembly  &  Test 
Integration  &  Ver. 

Build  2 

Design 

Assembly  &  Test 
Integration  &  Ver. 

Product  Integration 
Formal  Qual.  Test 


2002  2003 


Tool  tips: 

The  top  two  charts  were 
made  in  Excel  and 
manually  manipulated. 

The  Gantt  chart  can  be 
generated  using  any 
scheduling  software. 
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What  We  Learned 

From  Schedule  and  Progress  indicators 

•  cost  and  schedule  slippage  --  EV  chart 

•  activities  taking  longer  than  planned  --  Gantt  chart 

•  assembly  and  test  behind  schedule  --  components 
completion  chart 

What  does  this  mean? 

•  confirms  we  have  a  problem 
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Resources  and  Cost  Indicators 


Analysis/Probing  Questions 

•  Is  the  staff  allocation 
contributing  to  the 
problem  (too  many,  too 
few,  wrong  time  frame)? 

•  What  is  rate  of  staff 
turnover? 

•  How  does  actual  staff 
compare  to  planned  staff 
allocation? 
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Resources  and  Cost  Indicators 


Pro  Plar-"  2  3  3  3  0  9 

22 

23 

23 

22 

22 

22  2*v 

15 

u  ActikKS^  3  4  5  5  5  -B— 9 

15 

18 

19 

19 

20 

20^34r 

30 
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2  2 

5 

5 

7 

8 

8 

8  8 

5 

Tester  AcW  3  3  3  3  3  _ _ _  5 

5  5 

5 

5 

5 

5 

5 

5  5 

5 

Tool  tip:  This  chart  was  made  in  Excel  and  manually  manipulated. 
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What  We  Learned 

From  Resources  and  Cost  Indicators 

•  staffing  did  not  follow  planned  level 

-  too  many  at  beginning  of  project 

-  testers  and  programmers  used  to  fill  in  for  analysts 
and  designers  =>  high  re-training  costs 

-  high  turnover  rate  =>  training  &  getting  up-to-speed 
costs 

What  does  this  mean? 

•  cost  overrun  due  partly  to  staffing  problems 
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Growth  and  Stability  Indicators 


Analysis/Probing  Questions 

•  Are  the  requirements  stable? 

•  What  is  the  code  growth? 

•  Is  functionality  being 
transferred  from  build  1  to 
build  2?  If  so,  how  does  this 
effect  the  delivery  date? 
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Requirement  Changes  Information 


As  of  Jul  03 


2002  2003 


Tool  tip:  This  chart 
can  be  generated 
in  Excel  followed 
by  manual  editing 
using  the  drawing 
toolbar 


2002 

2003 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Dec 

Mar 

Jul 

Req  Changes 

10 

11 

6 

14 

10 

8 

4 

14 

4 

1 

5 

Complexity 

Low 

Low 

Low 

Low 

Low 

Low 

Low 

Low 

Low 

Low 

Low 

Resources 

(staff-days) 

4 

5 

3 

2 

4 

2 

3 

3 

2 

1 

2 
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Growth  and  Stability  Indicators 


Size  Growth  Requirements  per  Build 


Tool  tip:  This  chart  was  made  in  Excel  and 
manually  manipulated. 


Contractor’s  Explanation: 

•  Functions  deferred  to  later  build 
because  of  unanticipated 
complexity 
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What  We  Learned 

From  Growth  and  Stability  Indicators 

•  requirement  changes  are  of  low  complexity  but  will 
have  some  ripple  effect 

•  code  production  below  planned  value 

•  functionality  being  deferred  from  build  1  to  build  2 
attributed  by  contractor  to  unanticipated  complexity 

What  does  this  mean? 

•  expect  further  cost  and  schedule  growth  due  to  low 
code  production  and  increased  number  of  functions  to 
be  implemented  in  Build  2 

•  expect  an  impact  on  completion  date  due  to  functions 
deferred  to  Build  2 

•  expect  the  possibility  of  a  “Build  3”  proposal 
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Product  Quality  Indicators 


Analysis/Probing  Questions 

•  Are  the  defined  processes 
being  followed? 

•  What  is  the  rate  of  closure  for 
trouble  reports? 

•  What  type  of  trouble  reports 
are  being  detected?  In  what 
phase? 
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Product  Quality  Indicators 


J  FMAMJ  JASONDJ  FMAMJ  JASOND 

2002  2003 


Open  and  Closed  STRs 


Priority  Priority  Priority  Priority  Priority 
1  2  3  4  5 

Showstopper  Nit 


Tool  tip:  This  chart  was  made  in  Excel  and  manually  manipulated. 


Tallies  on  Jul  03 

□  Open  (200) 

■  Closed  (400) 
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Classifying  Trouble  Report  Defects 
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What  We  Learned 

From  Product  Quality  Indicators 

•  STRs  being  opened  faster  than  they’re  being  closed 

•  Code  inspections  should  have  found  defect  types 

What  does  this  mean? 

•  Code  inspection  process  allowed  large  number  of 
defects  to  slip  through. 
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Development  Performance 
Indicators 


Analysis/Probing  Questions 

•  Are  the  defined  processes 
being  followed? 

•  Are  any  defined  processes 
being  skipped? 
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Development  Performance 
Indicators 


Tool  tip:  This  chart  was  made  in  Excel  and  manually  manipulated. 
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What  We  Learned 

From  Development  Performance  Indicators 

•  adherence  to  defined  process  decreased  over  time 

•  stopped  doing  inspections 

What  does  this  mean? 

•  defects  usually  detected  during  code  inspections 
allowed  to  slip  through 

•  impact  on  cost  and  schedule  due  to  rework 
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Reasons  for  Slippage 

Staffing  problems: 

•  too  many  at  beginning  of  project 

•  below  planned  level  during  most  of  development 

-  noting  that  productivity  increased  dramatically 

•  high  turnover  rate 

Process  compliance: 

•  stopped  doing  inspections 

•  allowed  errors  to  leak  to  later  phases 

Requirements  changes  after  Build  2  code  and  unit  test 
Conclusion: 

•  expect  further  cost  and  schedule  growth  due  to  low  code 
production  and  increased  number  of  functions  to  be 
implemented  in  Build  2 
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Possible  Actions 

Developer  Actions 

•  replan  based  on  current  performance 

•  get  staffing  under  control 

-  verify  the  skills  balance  of  resources 

-  do  not  decrease  staffing  to  conform  to  “planned” 
staffing,  particularly  if  that  would  decrease  the 
number  of  programmers 

•  restart  inspections 

-  code 

-  test  cases 

Acquirer  Decision  Options 

•  use  contract  labor  (additional  costs) 

•  deliver  smaller  size  -  less  functionality 

•  accept  schedule  slip 
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Focusing  In 


Earlier: 

•  trends,  roles,  models 

•  measurement  methods 

•  evaluating  deliverables 

Key  Points: 

•  use  the  Performance  Analysis 
Model  as  a  causal  analysis 
navigation  aid 

•  always  use  multiple  indicators 

•  couple  data  analysis  with 
knowledge  of  your  and  your 
contractor’s  processes 


What’s  in  sight: 

•What  would  YOU  include  in  the  contract? 
•How  to  communicate  using  your  measures 
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Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Writing  Your  Contract 

Performance-based  contracting 

•  contractors  are  paid  based  on  how  they  meet 
predefined  metrics 

General  tips: 

•  Consider  project,  product,  process  measures 

•  Specify  frequency  of  reporting 

•  Specify  target  performance  where  known 

-  the  “SMART”  approach  applies:  specific,  measurable, 
attainable,  realistic,  timely 
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Discussion:  Write  Your  Contracts! 

For  the  two-contract  illustration  just  reviewed,  what 
measures  would  YOU  request  in  the  contracts? 


Which  measures  do  you 
think  would  be  readily 
available  (or  not)? 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  94 


Carnegie  Mellon 

Software  Engineering  Institute 


Outline 

Context 

•  state  of  the  community 

•  changing  perspectives 
Background 

•  roles  &  responsibilities 

•  maturity  models 

•  measurement  &  analysis  methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Sr.  Mgmt  dashboard 

•  quality  trends 

•  selected  project  EV  data 


Middle  Mgmt  dashboard 

•  system  documentation  and 
testing 


t 


Analysis  Indicators 

Reqts  completeness  - 
original,  at  inspection, 
approved  (for  contract  1 ) 


Goal: 

Establish  Acquisition 
Processes 

l 


Sr.  Mgmt  scorecard  ; 
Middle  Mgmt  dashboard 


Strategy  to  accomplish  goal 

•  Reference  models:  CMMI,  SA  CMM, 

ieee/iso  12207 

•  Leverage  CMMI  capabilities  built  in 
engineering:  MA,  REQM,  RD,  CAR 

•  Aim  for  CMMI  capability  in  selected  PAs: 
SAM,  DAR,  RSK,  PP/PMC,  CM,  PPQA 

•  Reference  all  SA-CMM  Level  2  kPAs, 
noting  overlaps  with  CMMI 


! _ 

Tasks  to  Accomplish  goal 

•  Implement  requirements  management 
process 

•  Tailor  existing  project  monitoring  processes 
for  acquisition  managers 


Success  Indicators 

process  owners,  training, 

CM,  and  documentation 
(future:  procedural  adherence) 


Middle  Mgmt  Dashboard 

•  selected  SPI  plan  EV  data 


Progress  Indicators 

start,  finish  dates 
with  progress  noted 
(move  toward  EV) 
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lustration:  Goal  Structure 
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Illustration:  Success  Indicators 
Establish  Acquisition  Processes 

Two  key  success  indicators  (excerpted  from  indicator  templates) 

•  status  of  ownership,  training,  documentation,  configuration 
management  of  processes  (evolve  into  procedural  adherence) 

•  status  of  training,  using  ISO12207  to  group  processes 


After  processes  established,  monitor  sustainment  or  adherence 
•  use  appraisal  and/or  audit  results 
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Illustration:  Senior  Management 
Reporting 

Required  contractor  metrics  reported  by  all  programs 

•  size  growth 

•  workforce  size  and  qualifications 

•  selected  earned  value  (EV) 

•  quality  trends 

•  requirements  fulfillment 

Required  acquirer  metrics  reported  by  all  programs 
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Outline 

Context 

•  State  of  the  community 

•  Changing  Perspectives 
Background 

•  Roles  &  Responsibilities 

•  Maturity  Models 

•  Measurement  &  Analysis  Methods 
Scenario 

•  goal-setting  and  success,  progress,  analysis  indicators 

•  inspecting  the  quality  of  deliverables:  requirements 

•  monitoring  and  oversight:  progress  analysis 

•  measurement  in  the  contract 

•  communicating  with  integrated  measures 
Summary 
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Roles  and  Information  Exchange 


Contractual  Handshake 
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Measuring  Project,  Product,  Process 


Processes  -s 


Acquirer 


Pre-award 

activities 

Post-award 

activities 


Project 

•Schedule 

(status,  projection,  trend) 
•Cost 

(status,  projection,  trend) 
•Requirements  satisfaction 


Contractual 

Handshake 


Exchange  of 
indicators  / 
information 
for  tracking, 
monitoring, 
direction,  etc, 


Supplier 


Develop, 

Customize, 

Integrate 

•  systems 

•  software 

•  COTS 


Products 


Relationship 

•Roles-Changes 

•Invoicing-payment 


•Supplier  Produced 
•Quality  (amount  of  rework) 

•Acquisition  Organization  Produced 
•Quality  (amount  of  rework) 
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Summary  -  Focus  Points 

Key  acquisition  responsibilities  (after  contract  award): 

•  monitoring  and  oversight 

•  inspecting,  reviewing,  and  understanding  documents 
and  other  work  products 

Post-contract  award  success  depends  on  pre-contract 

award  activities 

•  building  measurement  expectations  into  contracts 

•  establishing  good  partnerships  and  working 
relationships  with  contractors 

Measure  products,  processes,  projects,  relationships 

•  requirements  development,  management,  products 
should  not  be  exempt!  They  are  measurable. 
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Contact  Information 

Wolf  Goethert 

Software  Engineering  Institute 
Measurement  &  Analysis  Initiative 
Email:  wbg@sei.cmu.edu 
412-268-3889 

Jeannine  Siviy 

Software  Engineering  Institute 
Measurement  &  Analysis  Initiative 
Email:  imsiviv@sei.cmu.edu 
412-268-7994 

Robert  Ferguson 
Software  Engineering  Institute 
Measurement  &  Analysis  Initiative 
Email:  rwf@sei.cmu.edu 
412-268-9750 


©2003  by  Carnegie  Mellon  University 


Version  1.0 


page  104 


Carnegie  Mellon 

Software  Engineering  Institute 


References 

Note:  URLs  valid  as  of  tutorial  delivery  date. 
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Delivered  at  SEPG  2003,  Boston,  MA 
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Integration  (CMMIsm),  ICSQ,  29  October  2003,  Ottawa,  Canada 

[DZ  -  P  03] 

Adapted  from  [DZ  02]  by  Mike  Phillips  for  a  client  workshop 

[Ferguson  03] 

Ferguson,  Jack,  Use  of  CMMI  in  an  Acquisition  Context,  CMMI  Users  Group  2003, 

Denver,  CO 

[GQIM  96] 

Goal-Driven  Software  Measurement--A  Guidebook 
http://www.sei.cmu.edu/publications/documents/96.reports/96.hb.002.html 
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Note:  URLs  valid  as  of  tutorial  delivery  date. 
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http://itmanaqement.earthweb.com/erp/article.php/1558431 

Practical  Software  and  Systems  Measurement  A  Foundation  for  Objective  Project 
Management,  Guidebook,  version  4.0b,  Practical  Software  and  Systems 

Measurement  Support  Center,  U.S.  TRACOM-ARDEC,  AMSTA-AR-QA-A,  Picatinny 
Arsenal,  NJ,  Website:  www.psmsc.com,  October  2000 
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Reading  &  Resources  1 

Note:  URLs  valid  as  of  tutorial  delivery  date. 

Practical  Software  and  Systems  Measurement  (PSM) 

•  reference  for  the  Performance  Analysis  Model 

•  reference  lists  of  measures  to  consider 

•  http://www.psmsc.com 

Goal  Driven  Measurement  (GDM)  and 
Goal-Question-Indicator-Metric  (GQIM) 

•  front  end  for  selecting  most  relevant  PSM  measures 

•  used  for  developing  context-specific  indicators,  particularly 
“success  indicators” 

•  “Goal-Driven  Software  Measurement-A  Guidebook” 
http://www.sei.cmu.edu/publications/documents 

/96.reports/96.hb.QQ2.html 
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Reading  &  Resources  2 

Note:  URLs  valid  as  of  tutorial  delivery  date. 

Defense  Acquisition  University  (DAU)  Deskbook 

•  http://deskbook.dau.mil/isp/default.jsp 

•  provides  information  about  regulatory  references,  mandatory 
and  discretionary  references  by  service  branch,  and  several 
knowledge  repositories 

Guidelines  for  Successful  Acquisition  and  Management  of 
Software-Intensive  Systems, 

http://www.stsc.hill.af.mil/resources/tech  docs/index.html 

Acquisition  Centers  of  Excellence 

•  Air  Force,  for  instance  ESC  Hanscom 

-  http://esc.hanscom.af.mil/ESC-BP/ 

•  Navy 

-  http://www.ace.navy.mil/public/html/ 
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Reading  &  Resources  3 

Note:  URLs  valid  as  of  tutorial  delivery  date. 

Project  Management  Body  of  Knowledge  (PMBOK®) 

•  proven,  traditional  project  management  practices  and 
innovated,  advanced  practices  with  more  limited  use 

•  Project  Management  Institute  Guide  to  the  PMBOK  contains 
the  generally  accepted  subset  of  knowledge  and  practices 
that  are  applicable  to  most  projects  most  of  the  time 

-  http://www.pmi.org/info/PP  StandardsExcerpts.asp 

-  http://www.pmi.org/info/PP  PMBQK2000Excerpts.asp 
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